Constraint-Based Spatialization

This paper describes an application of constraint programming to interfaces for audio mixing. MidiSpace is an interface representing each sound source of a musical piece as a graphical icon, as well as an object corresponding to the listener in a window. MidiSpace is coupled to a spatialization system so that moving graphical objects modifies the audio mixing of the musical piece according to the respective position of the sound sources to the avatar. We further introduce a constraint-based mechanism which allows to maintain consistency in the overall mixing. Constraints represent properties of related sound sources, which should always remain true, and may be stated by the user through the interface. When an object is moved, a constraint solver uses the constraints to propagate changes. We describe the library of currently designed constraints, and propose an extension of the system to handle reproduction systems with multiple loudspeakers. 1 Music Spatialization Music spatialization has long been an intensive object of study in computer music research. Most of the work so far has concentrated in building software systems that simulate acoustic environments for existing sound signals. These works are based on techniques allowing to recreate impression of sound localization using a limited number of loudspeakers. For instance, The Spatialisateur IRCAM [4] is a virtual acoustic processor that allows to define the sound scene as a set of perceptive factors such as azimuth, elevation and orientation angles of sound sources relatively to the listener. This processor can adapt itself to any sound reproduction configuration, such as headphones, pairs of loudspeakers, or collections of loudspeaker. Other commercial systems with similar features have recently been introduced on the market, such as Roland RSS, the Spatializer (Spatializer Audio Labs) or Q-Sound labs’s Q-Sound, which builds extended stereophonic images. This tendency to propose integrated technology to produce 3D sound is further reflected, for instance, by Microsoft’s DirectX API now integrating 3D audio. Spatialization techniques have mostly been used to enhance existing interfaces or systems such as the Cave or CyberStage [2] [3]. Conversely, we are interested in building interfaces for controlling spatialization per se. The main applications are 1) high level interfaces for mixing devices, and 2) interfaces for future multi track listening devices. In this context, the main concern is to maintain some sort of consistency of musical pieces, while allowing the user to navigate freely in a control space. We will first describe our system MidiSpace, which precisely allows users to control in real time spatialization of sound sources, without any restriction. In Section 3, we will show how to add some semantics to limit the range of user actions in a meaningful way. Finally in Section 4 we will show how to extend the system to handle sound reproduction systems with several loudspeakers in an homogeneous way. 2 The MidiSpace System 2.1 The basic system MidiSpace is an interface for controlling an arbitrary spatialization system. The basic idea is to represent graphically sound sources in an editor, as well as an avatar that represents the listener itself. In this editor, the user may either move its avatar around, or move the instruments themselves. The relative position of sound sources and the listener’s avatar determine the overall mixing of the music, according to simple mapping functions, as illustrated in Figure 1. The 2D interface of MidiSpace is represented in Figure 2. The real time mixing of sound sources is realized by sending Midi volume and panoramic messages to the spatialization system. listener’s avatar sound source